The Cochrane mask fiasco: How the evidence-based medicine paradigm can produce misleading results

On Friday, the New York Times published on Opinion piece by Zeynep Tufekci entitled Here’s Why the Science Is Clear That Masks Work. Written in response to a recent Cochrane review, Physical interventions to interrupt or reduce the spread of respiratory viruses, that had over the last month been widely promoted by antimask and antivaccine sources, the article discusses the problems with the review and its lead author Tom Jefferson, as well as why it is not nearly as straightforward as one might assume to measure mask efficacy in the middle of a pandemic due to a novel respiratory virus. Over the month since the review’s publication, its many problems and deficiencies (as well as how it has been unrelentingly misinterpreted) have been discussed widely by a number of writers, academics, and bloggers, including our very own Steve Novella, as swell as Katelyn Jetelina and Kristen Panthagani, Gideon Meyerowitz-Katz Jason Abaluck, among others.

My purpose in writing about this kerfuffle is not to rehash (much) why the Cochrane review was so problematic. Rather, it’s more to look at what this whole kerfuffle tells us about the Cochrane Collaborative and the evidence-based medicine (EBM) paradigm it champions. AFter all, the authors cited above do an excellent job of that, although they do miss a lot of important context about the Cochrane Collaborative, Tom Jefferson, and the EBM paradigm under which Cochrane operates and that it has successfully promoted as the “gold standard” of scientific and medical evidence. I want to ask: What is it about Cochrane and EBM fundamentalists who promote the EBM paradigm as the be-all and end-all of medical evidence, even for questions for which it is ill-suited, that can produce misleading results? This is a topic that we at SBM used to write about on a regular basis when it came to the EBM paradigm’s application alternative medicine interventions, where EBM has never truly been able to definitively conclude that magical interventions—e.g., homeopathy, reiki, “energy healing” like therapeutic touch, and intercessory prayer—don’t work because the RCTs were “equivocal.” We used to call EBM’s failure to consider the low to nonexistent prior probability as assessed by basic science that magic like homeopathy could work its “blind spot.” Jefferson’s review, coupled with the behavior of EBM gurus like John Ioannidis during the pandemic, made me wonder if there’s another blind spot of EBM that we at SBM have neglected, one that leads to Cochrane reviews like Jefferson’s and leads EBM gurus like Ioannidis to make their heel turns so soon after the pandemic hit, which, truth be told, did catch me by surprise.

First, let’s look at how the review was used and misused. Much of how it was received was predictable, although, before Tufekci’s article, I had seen little discussion of Tom Jefferson himself. Let’s just say that it did not surprise me that he was the first author of the Cochrane mask review, for reasons that will soon become apparent.

The usual suspects do their usual thing with the Cochrane review

Before I “go meta”—not that Meta!—it’s necessary to emphasize that not even this not-so-great review ever actually concluded that “masks don’t work.” Rather, it only concluded that the evidence is uncertain. Naturally, not being about nuance, antimaskers referenced the Cochrane systematic review as definitive scientific evidence that “masks don’t work,” even to the point of being cited in a Congressional committee hearing to put CDC Director Dr. Rochelle Walensky on the defensive over mask mandates. Elsewhere, many of the usual suspects were Tweeting and posting proclamations like this:

I could go on and on, as there are thousands of Tweets like this, as well as likely thousands of posts on various other social media with variations of the very same message, specifically that the “gold standard” of evidence, a Cochrane systematic review, “proved” that “masks don’t work.” Indeed, perusing the triumphant gloating on social media from ideological sources opposed to COVID-19 interventions, including masks and vaccines, I was struck by how often they used the exact phrase “gold standard” to portray Cochrane as an indisputable source, all to bolster their misrepresentation. As Tufekci reports, this misrepresentation must have finally rattled Cochrane’s leadership, as they finally issued a statement:

Now the organization, Cochrane, says that the way it summarized the review was unclear and imprecise, and that the way some people interpreted it was wrong.

“Many commentators have claimed that a recently updated Cochrane review shows that ‘masks don’t work,’ which is an inaccurate and misleading interpretation,” Karla Soares-Weiser, the editor in chief of the Cochrane Library, said in a statement.

“The review examined whether interventions to promote mask wearing help to slow the spread of respiratory viruses,” Soares-Weiser said, adding, “Given the limitations in the primary evidence, the review is not able to address the question of whether mask wearing itself reduces people’s risk of contracting or spreading respiratory viruses.”

She said that “this wording was open to misinterpretation, for which we apologize,” and that Cochrane would revise the summary.

Soares-Weiser also said, though, that one of the lead authors of the review even more seriously misinterpreted its finding on masks by saying in an interview that it proved “there is just no evidence that they make any difference.” In fact, Soares-Weiser said, “that statement is not an accurate representation of what the review found.”

There is now a statement on the Cochrane website, as Tufekci notes:

There are also a number of other interesting things about this review, but first let’s look at why I was not in the least bit surprised to learn who the first author was. It also led me to wonder now, as I did in 2009 and 2013, why the Cochrane Collaborative still employs him.

Methodolatry at Cochrane, or: Meet the new Tom Jefferson, same as the old Tom Jefferson

We have met that lead author before. His name is Tom Jefferson, and we at SBM have been writing about him intermittently since 2009, as I will discuss later. What really stood out in the NYT article—and, given his history that I will relate— is that soon after his Cochrane systematic review was published Jefferson gave an interview to Maryanne Demasi, a journalist who has been over the years devolved into an antivaccine propagandist, as one can easily see merely by perusing her Substack, which is where she first published her interview with Jefferson, and how many times she is positively cited by antivax sources. Her interview with Jefferson was widely picked up by the usual suspects in the antivax social media ecosystem, such as the Brownstone Institute and Robert F. Kennedy, Jr.’s Children’s Health Defense. Unsurprisingly, Demasi is also a contributor to the Brownstone Institute website, as is Tom Jefferson himself, something that has only been rarely mentioned in the reporting about this review.

In the interview, Jefferson made very confident proclamations, like:

There is just no evidence that they [masks] make any difference. Full stop. My job, our job as a review team, was to look at the evidence, we have done that. Not just for masks. We looked at hand washing, sterilisation, goggles etcetera…

And:

There’s no evidence that they [masks] do work, that’s right. It’s possible they could work in some settings….we’d know if we’d done trials. All you needed was for Tedros [from WHO] to declare it’s a pandemic and they could have randomised half of the United Kingdom, or half of Italy, to masks and the other half to no masks. But they didn’t. Instead, they ran around like headless chickens.

First of all, as Steve Novella, Katelyn Jetelina and Kristen Panthagani, Gideon Meyerowitz-Katz, Jason Abaluck, and others have pointed out, the study doesn’t say that there is “no evidence” that facemasks “do work.”

I facepalmed when I read Jefferson’s simplistic (to the point of being simple-minded) pronouncement about randomized clinical trials (RCTs), wanting to respond: Tell me you don’t understand the ethics of RCTs without telling me you don’t understand the ethics of RCTs. Also, tell me you don’t understand the logistics of doing a randomized clinical trial this massive without telling me you don’t understand the logistics fo doing an RCT this massive. I will discuss these issues more near the end of this post, but first I need to set up the discussion.

There is a pattern that we at SBM had been observing about Jefferson for a very long time, in which he is much more circumspect with his conclusions in the systematic reviews he writes for Cochrane but then gives interviews to sympathetic journalists in which he makes pronouncements like the ones he made in his interview Demasi; for instance, during the last pandemic, H1N1 influenza in 2009, he all but came out and stated, “The flu vaccine doesn’t work.” It’s his schtick, apparently. Jefferson is much less definitive in his Cochrane conclusions and includes the necessary shortcomings and caveats, likely because peer reviewers insist on it in academic articles, but then says what he really thinks in interviews, where there is no peer review. Sometimes, he’s even appeared with utter quacks like Gary Null to falsely claim that flu vaccines aren’t safe in pregnancy and that statins don’t work.

I once referred to his schtick this way:

…he insinuates that the flu vaccine doesn’t work while never actually saying that you shouldn’t get it. He maintains plausible deniability, while trashing the vaccine left and right. What he says in public also differs from what he says in his Cochrane reviews, as I’ve pointed out before. For his reviews, he has to stick to the evidence, and peer reviewers have to be satisfied. When he’s talking to journalists, he can let his freak flag fly higher and, as I put it, go full mental negative on the flu vaccine.

Which brings us back to my purpose in writing this.

I had been meaning to write about the Cochrane systematic review on masks almost since it was first published, but somehow other things kept getting in the way, including a personal family health emergency that I mentioned six weeks ago that has stabilized but is ongoing. I simply didn’t have the time to delve deeply, and then when I finally did Steve had already written about the systematic review.

However, in the back of my mind, another angle that no one had covered (much) kept percolating. I’ve noticed over the last three years, this weekend having encompassed the third anniversary of the World Health Organization’s declaration of COVID-19 as a pandemic, a tendency for scientists who were known primarily before the pandemic as strong advocates of evidence-based medicine, devolving into promoters of COVID-19 denial, antimask, anti-public health, and even antivaccine pseudoscience. Think Dr. John Ioannidis, whom we at SBM lionized before 2020. Think Dr. Vinay Prasad, of whose work on medical reversals and calls for more rigorous randomized clinical trials of chemotherapy and targeted therapy agents before FDA approval we generally wrote approvingly.

Basically, what Jefferson exhibited in his almost off-the-cuff claim that massive RCTs of masks should have been done while a deadly respiratory virus was flooding UK hospitals was something we like to call “methodolatry,” or the obscene worship of the RCT as the only method of clinical investigation. I’ve long pointed out how methodolatry leads the EBM paradigm to be too open to alternative medicine, but it turns out that it might also lead to COVID contrarianism. So let’s take a trip down memory lane, back to the early days of this blog, before moving on to discussing methodolatry in the context of Jefferson’s studies and how many EBM

EBM and “complementary and alternative medicine” (CAM)

Longtime regular readers who have been with us since the beginning might remember that one common theme of this blog has been the difference between evidence-based medicine (EBM) and what we introduced as science-based medicine (SBM). Indeed, in Steve Novella’s very first post introducing the blog 15 years ago, he described the difference between EBM and SBM this way:

Within the practice of medicine there is already a recognition of the need to raise the standards of evidence and the availability of the best evidence to the practitioner and the consumer – formalized in the movement known as evidence-based medicine (EBM). EBM is a vital and positive influence on the practice of medicine, but it has its limitations. Most relevant to this blog is the focus on clinical trial results to the exclusion of scientific plausibility. The focus on trial results (which, in the EBM lexicon, is what is meant by “evidence”) has its utility, but fails to properly deal with medical modalities that lie outside the scientific paradigm, or for which the scientific plausibility ranges from very little to nonexistent.

We spent much of the first year of this blog’s existence, and, rather than reinvent the wheel yet again, I will quote liberally from some of these old posts. Indeed, a few weeks later, Dr. Kimball Atwood expanded on the concept of what we mean by “science-based medicine” as opposed to EBM:

Some might be surprised to find that EBM is not synonymous with “science-based medicine.” Although based on previous, evolving standards of clinical trial designs, statistics, epidemiological methods and other pertinent tools, EBM is a semi-formal movement within modern medicine that has existed for fewer than 20 years; it comprises sets of guidelines for assessing evidence, which will be discussed further below.

And:

To many in this era of EBM it seems self-evident that all unproven methods, including homeopathy, should be subjected to such scrutiny. After all, the anecdotal impressions that are typically the bases for such claims are laden with the very biases that blinded RCTs were devised to overcome. This opinion, however, is naive. Some claims are so implausible that clinical trials tend to confuse, rather than clarify the issue. Human trials are messy. It is impossible to make them rigorous in ways that are comparable to laboratory experiments. Compared to laboratory investigations, clinical trials are necessarily less powered and more prone to numerous other sources of error: biases, whether conscious or not, causing or resulting from non-comparable experimental and control groups, cuing of subjects, post-hoc analyses, multiple testing artifacts, unrecognized confounding of data due to subjects’ own motivations, non-publication of results, inappropriate statistical analyses, conclusions that don’t follow from the data, inappropriate pooling of non-significant data from several, small studies to produce an aggregate that appears statistically significant, fraud, and more.

A few years later, Dr. Atwood wrote a multipart article asking Does EBM Undervalue Basic Science and Overvalue RCTs? (Hint: the answer is yes.) He also wondered whether it is a good idea to test highly implausible health claims. (Hint: the answer is usually no, at least not using RCT efficacy trials of the sort placed near the very top of the EBM pyramid of evidence, just under systemic reviews, evidence syntheses, and meta-analyses of RCTs.)

As I like to ask: Which of the following is more likely, that a 30C homeopathic solution of…something…that has been diluted on the order of 10³⁷-fold more than Avogadro’s number and thus is incredibly unlikely to contain even a single molecule of that something has a therapeutic effect or that the RCTs concluding that it does reveal the problems and biases in clinical trials? As I also like to say, given the usual p-value of 0.05 designated for “statistically significant” findings, under ideal circumstances, with perfectly designed and executed RCTs, by random chance alone 5% of these RCTs will be “positive.” Of course, in the real world, RCTs are not perfect, either in design or execution, and the number of “false positives” is therefore likely considerably higher than 5%. Yet, basic science alone tells us that a 30C homeopathic remedy is indistinguishable from the water used to dilute it, which means a placebo-controlled RCT is testing placebo versus placebo and “positive” results show us nothing more than the noise inherent in doing RCTs.

I also like to caution: EBM is correct that “first principles” in basic science cannot tell you whether a treatment works or not. RCTs are indeed necessary. History is littered with treatments that appeared effective in preclinical in vitro and animal models but failed to pan out in clinical trials. However, in contrast, if basic science tells you that a treatment modality is impossible, that (like homeopathy) it would require that multiple well-established laws and theories of chemistry and physics be not just wrong, but spectacularly wrong, in order for it to “work,” then that should be enough. RCTs should not be necessary. Indeed, they would arguably be unethical, because another ethical requirement for clinical trials is that the treatment being tested be grounded in good science based on what is known at the time of the trial.

But what about COVID-19? Guess what? What has happened since the pandemic hit suggests to me that EBM also undervalues epidemiology compared to RCTs.

EBM during the pandemic: A different flavor of methodolatry?

Let’s revisit what Jefferson said in his interview with Demasi, because it’s important to my discussion of methodolatry. Basically Jefferson’s statement saying that the WHO should have just done RCTs of masks in the UK and Italy was perhaps the most blatant manifestation of “methodolatry,” a term used to describe the EBM tendency to view RCTs as the only valid method of clinical investigation and to devalue to an inappropriate degree any evidence that does not come from an RCT, that I can recall having seen in a long time. Indeed, Jefferson’s blithe suggestion of “Why didn’t they just do RCTs of masks” as a pandemic crashed into two major countries demonstrates just how methodolatry ignores practicality. (So easy!) It ignores expense, as Jefferson did when he suggested that RCT. It ignores ethics, as Jefferson did when he actually suggested an RCT of masks for a respiratory virus at the beginning of a pandemic that was flooding hospitals in Italy and the UK with very ill and dying patients, overwhelming the healthcare resources of the two countries.

Tufekci notes this problem in her article:

Why aren’t there more randomized studies on masks? We could have started some in early 2020, distributing masks in some towns when they weren’t widely available. It’s a shame we didn’t. But it would have been hard and unethical to deny masks to some people once they were available to all.

Actually, given the prior probability based on basic science that masks can block the respiratory droplets that transmit viruses like SARS-CoV-2, it is arguable that even what Tufekci proposed would have been unethical, as there would have been no clinical equipoise, an absolute requirement for an RCT to be ethical. Basically, as I’ve discussed before, clinical equipoise is genuine uncertainty whether an intervention is likely to help the experimental group compared to a control group not receiving it. As Tufekci notes:

Scientists routinely use other kinds of data besides randomized reviews, including lab studies, natural experiments, real-life data and observational studies. All these should be taken into account to evaluate masks.

Lab studies, many of which were done during the pandemic, show that masks, particularly N95 respirators, can block viral particles. Linsey Marr, an aerosol scientist who has long studied airborne viral transmission, told me even cloth masks that fit well and use appropriate materials can help.

Also, once again, carrying out RCTs of masks on a general population is incredibly difficult, which usually leaves epidemiological evidence as the best available evidence, which EBM claims to be all about:

Perhaps the best evidence comes from natural experiments, which study how things change after an event or intervention.

Researchers at Mass General Brigham, one of Harvard’s teaching hospital groups, found that in early 2020, before mask mandates were introduced, the infection rate among health care workers doubled every 3.6 days and rose to 21.3 percent. After universal masking was required, the rate stopped increasing, and then quickly declined to 11.4 percent.

In Germany, 401 regions introduced mask mandates at various times over three months in the spring of 2020. By carefully comparing otherwise similar places before and after mask mandates, researchers concluded that “face masks reduce the daily growth rate of reported infections by around 47 percent,” with the effect more pronounced in large cities and among older people.

Also note Jefferson’s dismissive tone, how he characterizes doctors and public health officials as “running around like headless chickens” because evidence was uncertain. Now, don’t get me wrong. Mistakes were made, as they say, in the heat of the initial confrontation with the novel coronavirus. Hydroxychloroquine was embraced by many hospitals in the US (including one where I worked and another major one in my city) as a standard of care in March 2020 based on very sketchy evidence and little more than the word of the Chinese government that it appeared to work, only to be found not to work as the pandemic’s first summer was drawing to a close.

Still, I’d be willing to bet that Jefferson belongs to what SBM contributor Dr. Jonathan Howard likes to call the “laptop class,” a term borrowed from Great Barrington Declaration (GBD) co-author Dr. Jay Bhattacharya when he railed against pandemic restrictions that in reality describes Dr. Bhattacharya himself, as well as his co-authors and nearly all of the GBD promoters. Let’s just say that I doubt that Dr. Jefferson went anywhere near COVID-19 wards to treat patients, as Dr. Howard did in New York City early in the pandemic. His “laptop class” status leads me to wonder what Dr. Jefferson would have recommended in March 2020, had he had actual “skin in the game” rather than just an intellectual interest in what EBM shows regarding pandemic restrictions?

Actually, we have an idea, given that in April 2020, Jefferson co-authored an article arguing tha there was no point in nonpharmaceutical interventions (NPIs) and mitigations to slow the spread of COVID-19 and railed against “lockdowns”:

Because there are no licensed treatments for COVID-19 non-pharmaceutical interventions, management of complications and early recognition of those deteriorating and most likely to benefit from hospitalisation, should be the mainstay of management.

Changing the emphasis from hospitals to the community could avert a disaster for the wider population. Care in the home setting restricts movements of the infected. All those with a fever and a cough should stay at home; they could be prescribed pulse oximeters, and oxygen could be delivered to severely affected cases; rescue antibiotics prescribed along with daily video-monitoring could be used to detect deterioration. In the older population, the mildly ill and the recovering, food supplies should be delivered at home.

Older Patients admitted to hospital are at greater risk of delirium, pressure sores, adverse effects of new medications, malnutrition and hospital-acquired infections. [6] An older person admitted to hospital runs the risk of never seeing the light of day again. This is probably the clearest message coming from Italy.

Lockdown is going to bankrupt all of us and our descendants and is unlikely at this point to slow or halt viral circulation as the genie is out of the bottle. What the current situation boils down to is this: is economic meltdown a price worth paying to halt or delay what is already amongst us?

Masks, I note, are not “licensed,” nor do they require a license. Also note Jefferson’s complete disconnect from the actual dire situation “on the ground” in April 2020. He suggests these policies as though they could easily be implemented immediately and would not require considerable planning of weeks to months. In addition, one can already detect a significant bias against masks and other NPIs in favor of a “let ‘er rip” fatalism that six months later became the basis of the GBD. That same month, he questioned whether COVID-19 pandemic was actually a pandemic at all, rather than just a long respiratory illness season, arguing that because during a pandemic the proportion of deaths among the young should increase COVID-19 couldn’t be a true pandemic. I kid you not. He concluded:

The data support the theory that the current epidemic is a late seasonal effect in the Northern Hemisphere on the back of a mild ILI season. The age structure of those most affected does not fit the evidence from previous pandemics.

The outbreak does, however, fit with the WHO’s definition of a Pandemic. This definition does not help explain the age structure of those most affected, and how this differs from that of seasonal outbreaks. The definition of a pandemic remains elusive.

What would Sherlock have made of all this?

I daresay that Sherlock Holmes would have sharply and sarcastically criticized Jefferson’s “reasoning,” such as it is, the way he often did with his friend Watson’s attempts to figure out a case. Seriously, if you were arguing in April 2020 that COVID-19 was just a bad “seasonal” flu-like illness, you lost the plot three years ago.

Methodolatry beyond Cochrane and Jefferson: When is epidemiology sufficient?

Jefferson is not alone among EBM gurus who took a major heel turn during COVID-19 to turn into contrarians who claim that NPIs like masks don’t work and that the COVID-19 vaccine should not have been authorized for children under an emergency use authorization because, supposedly, COVID is “not a danger” to children, even though around 1,800 children—a very large number for pediatrics!—have died of COVID since the pandemic hit, making it a leading cause of pediatric death. It is a number that is, on a population basis, roughly the same as the 400-500 deaths from measles that occurred every year before the first measles vaccine was licensed in the early 1960s. Back then, nearly every physician correctly believed that several hundred pediatric deaths a year were such a bad thing that the toll justified developing a vaccine and instituting the mass vaccination campaign in the 1960s that ultimately reduced the number of measles deaths a year to zero or single digits. Since COVID-19, a number of EBM fundamentalists think that a similar death toll among children is acceptable until and unless there are massive RCTs testing vaccination and then boosters in children.

Indeed, these EBM doctors calling for RCTs über alles don’t seem to realize that they are recycling exactly the same arguments that antivaxxers used to make against the measles vaccine, or vaccines against chickenpox or whooping cough, before the pandemic, all in the name of calling for ever larger, more impractical, and expensive RCTs of every conceivable combination of vaccination and boosters. Does any of this sound familiar? Those of us who remember countering the antivaccine movement before the pandemic remember antivaxxers routinely calling for huge, expensive, and impractical RCTs of variants of the entire vaccine schedule in order to determine whether vaccines cause autism or all the other things that they blame vaccines for. I’m not saying that EBM fundamentalists like Jefferson and Prasad are antivaxxers (although they certainly echo longstanding antivax tropes). What I am saying is that there seems to be something about EBM that leads its most visible and vocal promoters down a path towards sounding like antivaxxers. I realize that my saying this is likely to piss some people off, likely people I even respect, but it is an observation that has been increasingly difficult for me to discount.

Again, let’s go back to the evidence pyramid of EBM:

We used to routinely note that basic science considerations are relegated to the very bottom of the pyramid, as “level 5 evidence” consisting of “expert opinion without explicit critical appraisal, or based on physiology, bench research or ‘first principles.'”” It is true that epidemiology is higher on the pyramid—just below RCTs!—but for some reason, among EBM fundamentalists, that doesn’t seem to translate to any more respect than EBM gives basic science when it considers RCTs of homeopathy or energy medicine. Indeed, Jonathan Howard has documented how the response of self-proclaimed EBM adherents to proposed policies to vaccinate or boost children is always to say that there are “no RCTs” and to call for huge, impractical RCTs before recommending such interventions.

Indeed, look at Vinay Prasad writing at the Brownstone Institute and his Substack, you’ll see a similar attitude, although does give a slight amount of leeway:

1. In an emergency situation, if governments mandate or advise individual level behavioral interventions (e.g. masking), those entities should have generated robust data in 3 months (cluster RCTs) to demonstrate efficacy, or the intervention is automatically revoked. Some may argue 3 months is too short, but if it is truly a crisis warranting emergency proclamations, then you should see a signal in 3 months, and governments can expand sample size to ensure prompt results.
2. If a trial is positive that does not mean the policy continues forever, but must be debated (net benefit/ net harms/ tradeoffs) by the body politic.

Three months to generate robust RCT evidence? You can tell that Dr. Prasad has probably never actually gotten a clinical trial protocol funded and approved by an institutional review board (IRB) as principal investigator. Even under emergency situations, you might, if you’re lucky, be managing to randomize in 2-3 months. You won’t have robust evidence of anything yet. This is methodolatry at its most ridiculous.

Does EBM attract or contribute to contrarianism?

Don’t get me wrong here. The EBM paradigm is not wrong. Rather, it is seriously limited by, yes, methodolatry. Its entire framework assumes that RCTs are at the very strongest form of clinical evidence, thus putting aggregations of RCTs (e.g., meta-analyses, Cochrane systematic reviews, and other forms of systematic reviews) at the very top of the EBM pyramid. There is no doubt that EBM is correct that, in general, the best way to determine the efficacy of a drug or vaccine is a well-designed RCT. Far be it from me to imply otherwise! Moreover, I have echoed calls for more scientific rigor in drug approval, in particular the RCTs used to support the drug approval.

There are caveats, of course. the entire EBM model assumes that there is scientific plausibility. (Remember, that was one of the problems with EBM that we used to point out all the time with respect to clinical trials of homeopathy that could lead to misleading results.) In addition, it assumes that such a trial would be ethical. If one wanted to contest my criticism here, one could argue that EBM fundamentalists like Jefferson ignore the absolute requirement that a clinical trial be ethical and the relative requirement that it be feasible in favor of their worship of the RCT as the only valid method of clinical investigation that can produce definitive results.

Moreover, Cochrane meta-analyses are notoriously tricky to do. First of all, you have to decide what the question is. Then you have to decide what studies out there address the question. Then, for the case of something like masks, you have to realize that you will, by the time you go through the literature and apply EBM standards to them, looking the highest quality RCTs, you will be looking at only a small, highly select part of the literature. While this is fine for new drugs and asking if they are efficacious, for a multifactorial question like whether masks slow the spread of COVID-19 at the population level, a large amount of compelling and high quality scientific evidence could well be excluded because the EBM paradigm demands it.

Yet, as I like to point out, no EBM adherent seriously doubts that smoking and tobacco can cause lung cancer, heart disease, and a host of other maladies, even though there is not and never will be an RCT testing whether smoking causes these conditions because such a study would be profoundly unethical. Epidemiology is sufficient to lead to a conclusive conclusion that smoking causes lung cancer, for instance. Similarly, although there are RCTs that are cited as evidence that vaccines don’t cause autism, in reality it is nearly all epidemiology that leads to the conclusion that vaccines do not increase the risk of autism. No RCTs are required to reach these conclusions in a scientifically rigorous fashion, as is the case for a large number of other medical questions.

Even Cochrane seems to accept a lesser level of evidence in the midst of a deadly pandemic:

Brown, who led the Cochrane review’s approval process, told me that mask mandates may not be tenable now, but he has a starkly different feeling about their effects in the first year of a pandemic.

“Mask mandates, social distancing, the other shutdowns we had in terms of even restaurants and things like that — if places like New York City didn’t do that, the number of deaths would have been much higher,” he told me. “I’m very confident of that statement.”

Based on what, I wonder? There were no RCTs of mask use against COVID-19. So how could he know? I’m not saying he was wrong, but it would appear that even Cochrane editors abandon the strict EBM paradigm in the breach.

It occurred to me as I was wrapping up that one potential shortcoming of my argument is that so many of these methodolatrists tend to be very selective what conclusions to them require RCTs and what conclusions can be accepted without them. I cite again Vinay Prasad, who is very dismissive of epidemiological evidence for assessing the effectiveness of, for example, masks to prevent the spread of COVID-19 or of vaccination and/or boosters for COVID-19 in children, but will accept crappy dumpster dives into the Vaccine Adverse Event Reporting System (VAERS) database and various other epidemiological studies to conclude that myocarditis from the vaccines in adolescents and young men is an unacceptable risk.

And yet…I don’t think that ideological misuse of the EBM paradigm is the explanation. It is undoubtely true that some of these EBM fundamentalists do indeed do exactly that, calling for impossibly rigorous RCTs for interventions that they do not support while accepting much lower quality evidence to fear monger about low probability complications of those interventions. However, I can’t help but wonder if there is something about the EBM paradigm that either attracts contrarians or facilitates those predisposed to contrarianism to descend further into it. EBM certainly emphasizes uncertainty in evidence and conclusions, which is a good thing for skeptics to do when the question is a new drug. However, this skepticism can become unhealthy and misleading when coupled with the attitude, “If it’s not an RCT, it’s crap” or “Anything other than RCT evidence is crap.” EBM adherents will vigorously deny that they think that, but the conclusions coming out of Cochrane and touted by its most zealous adherents suggest otherwise.

Fifteen years ago, when Steve Novella co-founded SBM with Kimball Atwood, Wally Sampson, Harriet Hall, and me, I had to be convinced that the EBM model valued RCT evidence over everything and therefore discounted basic science. Once I realized that this was correct, i saw examples everywhere, mainly in the world of CAM. What I never suspected until now is that that same blind spot seems to contribute to an attitude that we can’t know anything with sufficient certainty to act, even in the middle of a pandemic.

I also note that 13 years ago, we noted that Cochrane might be finally starting to “get it” with respect to SBM as opposed to EBM when some EBM bigwigs actually started to concede that further studies might not be necessary for interventions like homeopathy. Unfortunately, I’ve seen little evidence that this “getting it” has stuck. Now I think that the evidence pyramid has had another malign effect during the pandemic, the devaluation of what is often the form of evidence most suited to address relevant questions given resources, medical ethics, and practicality, questions such as whether masks and other NPIs decrease the spread of COVID. Basically, more and more EBM seems to have one tool, and you know what they say about having only one tool and that tool is a hammer. The result was the Cochrane mask fiasco.

Fifteen years ago, we introduced the concept of SBM, primarily as a “fix” to EBM that would restore the consideration of basic science and the prior probability it estimates for highly implausible treatments. Based on what I’ve seen from EBM thought leaders during the pandemic, I’m wondering if another fix is in order from SBM. I will admit that I do not yet know what that fix will be yet, but I will be contemplating the question.